Method and device for partially unmasking an object in a video stream

ABSTRACT

A method for partially unmasking an object in a video stream comprises: displaying a first video stream in which objects are covered by privacy masks; receiving user input regarding a selection of an object being covered by a privacy mask, wherein the object has a first portion being associated with a first privacy level and a second portion being associated with a second, higher, privacy level; checking that a permission associated with the user input grants access to video data depicting object portions being associated with the first privacy level; giving access to video data depicting the first portion of the object; and displaying a second video stream which differs from the first video stream in that it includes the video data depicting the first portion of the object, while the second portion of the object is still covered by a privacy mask.

TECHNICAL FIELD

The present invention relates to the field of privacy masking of video.In particular, it relates to a method and a device for partiallyunmasking an object in a video stream.

BACKGROUND

As video surveillance of public spaces becomes more and more common, thefocus on protecting people's integrity increases. One way of protectingthe integrity of objects in a video is to conceal the objects withprivacy masks. A privacy mask is a masked area that conceals parts ofthe video from view, thereby protecting integrity of the concealed partof the video. The privacy mask may, for example, blur, pixelate orsolidly cover the masked area.

During normal operation, the privacy masked video is typically shown toan operator. However, when an incident occurs, an authorized operatormay request access to a non-masked version to see the objects withoutprivacy masks. A drawback with this approach is that it will not onlyreveal private information for the objects being involved in theincident, but also private information regarding other objects which arenot involved in the incident. Accordingly, private information ofinnocent objects is revealed without any reason. There is thus room forimprovements.

SUMMARY

In view of the above, mitigating the above problems and providing a wayof unmasking objects in video in a way that does not reveal more privateinformation about the objects than necessary in a given situation wouldbe beneficial.

According to a first aspect, a method for partially unmasking an objectin a video stream, comprises:

displaying a first video stream in which one or more objects are coveredby privacy masks,

receiving user input regarding a selection of an object being covered bya privacy mask in the first video stream, wherein the object has a firstportion being associated with a first privacy level and a second portionbeing associated with a second, higher, privacy level,

checking that a permission associated with the user input grants accessto video data depicting object portions being associated with the firstprivacy level,

giving access to video data depicting the first portion of the object,and

displaying a second video stream which differs from the first videostream in that it includes the video data depicting the first portion ofthe object, while the second portion of the object is still covered by aprivacy mask.

With the above method, an operator is first presented with a videostream in which objects are covered by privacy masks. In response toselecting an object in the video stream, an authorized operator isinstead presented with a video stream in which the selected object ispartially unmasked. This has several advantages. Firstly, by onlyunmasking the selected object, private information regarding the otherobjects is not revealed. Secondly, by partially unmasking the selectedobject, a portion of the selected object having a lower privacy levelmay be revealed while a portion of the object having a higher privacylevel may remain concealed so as to keep protecting the more sensitiveprivate information regarding the selected object. In many cases, it isenough to reveal video data depicting the object portion having thelower privacy level in order to analyze an incident. Particularly, thelower privacy level video data may be enough to see the actions of asuspected object and it is therefore not necessary to reveal higherprivacy level video data before a suspicion is verified. Consider theexample of video surveillance of a shop, where a privacy masked objectis suspected of shoplifting. To deem whether or not shopliftingoccurred, it is typically enough to unmask the body, or even only thearms and hands, of the suspected object. Revealing the face of thesuspected object, and thereby its identity, is not necessary until thesuspected action has been verified.

In view of the above, the method thus enables selective and partialunmasking of objects. In that way, private information regarding objectsin a video is not unnecessarily revealed. Further, by checking thepermissions of the user, the method makes sure that only authorizedusers are allowed to access the partially unmasked video. In particular,the permissions of the user are linked to the privacy levels of thedifferent portions of the object. Thus, a first user may not bepermitted to access video data depicting any portion of the object. Asecond user may be permitted to access video data depicting objectportions of the first privacy level. A third user may be permitted tofurther access video data depicting object portions of the second,higher, privacy level. The permissions are thus set on an object portionlevel rather than an object level, thereby allowing access control ofvideo data on an object portion level.

An object may have a plurality of object portions. Particularly, it mayhave a first portion and a second portion. In one example, the firstportion corresponds to the body (except for the face) of a person andthe second portion to a face of the person. In another example, thefirst portion corresponds to the body (except for the license plate) ofa car and the second portion to the license plate of the car. It isunderstood that an object may have more than two portions. The objectportions are assumed to be non-overlapping.

The object portions may be associated with different privacy levels. Bya privacy level of an object portion is generally meant how sensitivethe object portion is from an integrity point of view. For example, theprivacy level may indicate how easy it is to identify the object basedon image data depicting the object portion. The privacy levels of theobject portions may form a hierarchy of privacy levels reaching from alowest privacy level to a highest privacy level.

A privacy mask is a masked area that conceals parts of a video fromview. The purpose of the privacy mask is to protect the integrity of theconcealed part of the video. The privacy mask may be of any kind knownin the art. For example, it may be a contour mask following the contourof the object, a rectangular mask, a pixelated mask, a Sobel mask, asolid mask, etc.

By a privacy mask covering an object or an object portion is meant thatthe privacy mask conceals an image area associated with the object orthe object portion. In particular, at the outset of the method, theprivacy masks at least cover the first and the second object portions ofthe one or more objects. In other words, they cover object portionsbeing associated with the first privacy level and any higher level ofprivacy. It may therefore be said that the privacy masks cover theobjects to an extent necessary for a current privacy level to befulfilled. The privacy mask is dynamic. Particularly, as the objectmoves, the image area corresponding to the object moves and thereby alsothe privacy mask.

By a selection of an object is meant that at least one object isselected. It may hence be the case that more than one object isindicated by the user input.

By video data depicting an object portion is meant video data thatdepicts the object portion when not being covered by a privacy mask.

It is further noted that the second video stream typically only differsfrom the first video stream in that it includes the video data depictingthe first portion of the object.

By a permission associated with a user input is generally meant apermission or authorization of a currently logged-in user making theuser input. By the permission granting access to video data depictingobject portions being associated with a certain privacy level isgenerally meant that the currently logged-in user has the permission toget access to the video data depicting object portions having thatcertain privacy level.

The video data depicting the first portion of the object may beencrypted, wherein access to the video data depicting the first portionof the object is given by decrypting the video data depicting the firstportion of the object using a first decryption key associated with thepermission that grants access to video data depicting object portionsbeing associated with the first privacy level. The access to the videodata depicting the first portion of the object may hence be controlledby means of decryption. Specifically, only a user that has a permissionto access the right decryption key will be able to see the partiallyunmasked video.

The method may further comprise generating the second video stream bymerging the first video stream with the video data depicting the firstportion of the object. In that way, a video stream may be generated inwhich all non-selected objects are covered by privacy masks, as in thefirst video stream, while the first portion of the selected object isvisible.

The video data depicting the first portion of the object may be accessedfrom a video stream in which object portions associated with a firstprivacy level are visible and object portions associated with the secondprivacy level are covered by privacy masks. A video stream in which allthe objects are partially unmasked may hence be used as input. Bycombining that partially unmasked video stream, with the fully maskedfirst video stream it is possible to generate the second video stream inwhich only the selected object is partially unmasked. For example, thesecond video stream may be generated by replacing an image region whichincludes the object being covered by a privacy mask in the first videostream by a corresponding image region in the video stream in whichobject portions associated with a first privacy level are visible andobject portions associated with the second privacy level are covered byprivacy masks.

The user input may select an object in various ways. According to anembodiment, the user input regarding a selection of an object may beindicative of an image region in the first video stream, the imageregion including the object being covered by a privacy mask in the firstvideo stream. For example, the user input may draw an outline of aselected image region, such as by drawing the outline of a rectanglearound the selected object. This provides an easy way of selecting anobject. Further, it puts low requirements on the system since the systemdoes not need to be aware of which image regions that corresponds toprivacy masked objects. That input is instead given by the user in thiscase.

In an alternative embodiment, the first video stream includes metadatabeing indicative of image regions in the first video stream beingcovered by privacy masks, wherein the user input regarding a selectionof an object is indicative of an image position within one of the imageregions. For example, the user input may be in the form of a mouse clickon an object. In this case the metadata thus indicates which imageregions in the first video stream that corresponds to privacy maskedobjects. This is advantageous in that the metadata allows the method toderive what image region in the first video stream corresponds to theselected object as long as the user input specifies an image positionwithin one of the privacy masked objects.

As described above, the method allows an authorized user to selectivelyand partially unmask an object in a video. In some cases, theselectively and partially unmasked video may show that the selectedobject is innocent. In such cases there is no need to further unmask theselected object. However, in other cases, the selectively and partiallyunmasked video may show that the selected object is guilty of anincident. In those cases, it could be desirable to further unmask theselected object. For that purpose, the method may further comprise:checking that the permission associated with the user input furthergrants access to video data depicting object portions being associatedwith the second privacy level, giving access to video data depicting thesecond portion of the object, and displaying a third video stream whichdiffers from the first video stream in that it includes both the videodata depicting the first portion of the object and the video datadepicting the second portion of the object. Accordingly, a user beingpermitted to also view object portions of the second, higher privacylevel, is allowed to further unmask the selected object to reveal objectportion of the higher privacy level. The step of displaying the thirdvideo stream may be made in response to a further user input. Thus, thefurther unmasking is not performed until the user actively indicatesthat he wants to do so.

This may be extended to situations where there are more than two privacylevels ordered in a hierarchy. First, the user is presented with a fullymasked video, corresponding to a lowest privacy level in the hierarchy.Responsive to selection of an object, a permitted user is presented withvideo in which the selected object is partially unmasked to revealobject portions of the following, higher, privacy level in thehierarchy. An even further permitted user may then be presented with avideo in which the selected object is further unmasked to also revealobject portions of the following, even higher, privacy level in thehierarchy, etc. Accordingly, the selected object may be unmasked,privacy level by privacy level, as long as the permission of the userpermits.

Also, in this case the access to video data depicting the second objectportion may be controlled by way of decryption. Specifically, the videodata depicting the second portion of the object may be encrypted, andaccess to the video data depicting the second portion of the object maybe given by decrypting the video data depicting the second portion ofthe object using a second decryption key associated with the permissionthat further grants access to video data depicting object portions beingassociated with the second privacy level.

The third video stream may be generated by merging the first videostream with the video data depicting the first portion of the object andthe video data depicting the second portion of the object. In that way,a video stream may be generated in which all non-selected objects arecovered by privacy masks, as in the first video stream, while the firstportion and the second portion of the selected object are visible.

The video data depicting the second portion of the object may beaccessed from a video stream in which object portions associated with atleast the second privacy level are visible. For example, the videostream may include video data depicting both the first object portionsand the second object portions of the objects, such as in a fullyunmasked stream. In another example, the video data depicting the secondportion of the object is accessed from a video stream in which onlyobject portions associated with the second privacy level are visible. Inthe latter case, the third video stream may be a combination of threedifferent streams; a fully masked video stream, a video stream whichincludes video data depicting the first object portions, and a videostream which only includes video data depicting the second objectportions.

According to a second aspect, a device for partially unmasking an objectin a video stream comprises circuitry configured to execute:

a display function configured to display a first video stream in whichone or more objects are covered by privacy masks,

a user input receiving function configured to receive a user inputregarding a selection of an object being covered by a privacy mask inthe first video stream, wherein the object has a first portion beingassociated with a first privacy level and a second portion beingassociated with a second, higher, privacy level,

a permission checking function configured to check that a permissionassociated with the user input grants access to video data depictingobject portions being associated with the first privacy level,

a video data accessing function configured to give access to video datadepicting the first portion of the object, and

a display function configured to display a second video stream whichdiffers from the first video stream in that it includes the video datadepicting the first portion of the object, while the second portion ofthe object is still covered by a privacy mask.

According to a third aspect, a non-transitory computer-readable storagemedium comprises computer-code instructions adapted to carry out themethod of the first aspect when executed by a device having processingcapability.

The second and third aspects may generally have the same features andadvantages as the first aspect. It is further noted that all possiblecombinations of features are possible unless explicitly statedotherwise.

BRIEF DESCRIPTION OF THE DRAWINGS

The above, as well as additional objects, features and advantages, willbe better understood through the following illustrative and non-limitingdetailed description of embodiments, with reference to the appendeddrawings, where the same reference numerals will be used for similarelements, wherein:

FIG. 1 schematically illustrates a system for partially unmasking anobject in a video stream according to a first group of embodiments.

FIG. 2 schematically illustrates a device for partially unmasking anobject in a video stream according to the first group of embodiments.

FIG. 3 schematically illustrates a system for partially unmasking anobject in a video stream according to a second group of embodiments.

FIG. 4 schematically illustrates a device for partially unmasking anobject in a video stream according to the second group of embodiments.

FIGS. 5A-5D illustrate example video frames of video streams which areinput to the device of FIG. 2.

FIG. 6 is a flow chart of a method for partially unmasking an object ina video stream.

FIGS. 7A-7D illustrate example video frames of video streams which aredisplayed or generated by the device of FIG. 2 in example embodiments.

FIGS. 8A-8D illustrate example video frames of video streams which aredisplayed or generated by the device of FIG. 4 in example embodiments.

DETAILED DESCRIPTION

The present teachings will now be described more fully hereinafter withreference to the accompanying drawings, in which different embodimentsare shown.

FIG. 1 illustrates a system 100 in which example embodiments may beimplemented. The system 100 comprises a camera 102 and a device 104 forpartially unmasking an object in a video stream. The device 104 may formpart of a video management system via which a user can manage videocaptured by the camera 102. For example, a user may view video andprovide user input relating to the video via the device 104. Inparticular, a user may, via device 104, manage privacy masks beingincluded in the video.

The camera 102 of system 100 is configured to output at least threevideo streams 106, 108, 110. The output video streams 106, 108, 110differ from each other to the extent that objects depicted in the videoare covered by privacy masks. Particularly, the objects may be coveredby privacy masks to a larger extent in video stream 106 than in videostream 108. Further, objects may, in turn, be covered by privacy masksto a larger extent in video stream 108 than in video stream 110. Forexample, as shown in FIG. 5A, one or more objects 502-1, 502-2, 502-3may be fully covered by privacy masks in the video stream 106. In thevideo stream 108, the one or more objects 502-1, 502-2, 502-3 may bepartially covered by privacy masks, as shown in FIG. 5B. Further, in thevideo stream 110, there may be no privacy masks covering the one or moreobjects 502-1, 502-2, 502-3 as illustrated in FIG. 5C. As analternative, the video stream 110 may only include video data depictingthe portions 506-1, 506-2, 506-3 of the objects being covered by privacymasks in the video stream 108, as shown in FIG. 5D. In the latter case,a fully unmasked video may be generated by combining video streams 108and 110.

The video streams 106, 108, 110 may further each include metadata. Themetadata may, for each video frame in the stream, indicate the imageregions being covered by privacy masks in the respective video stream106, 108, 110. Thus, the metadata of the video stream 106 may indicatethree image regions corresponding to the objects 502-1, 502-2, 502-3.Similarly, metadata of the video stream 108 may indicate three imageregions corresponding to the object portions 506-1, 506-2, 506-3 thatare covered by privacy masks. The metadata may further include objecttracking metadata. The object tracking metadata associates each imageregion being covered by a privacy mask with an object identifier. Theobject tracking metadata enables keeping track of the movements of theprivacy masked image regions over time.

Video data may be associated with different levels of privacy. Videodata being associated with a higher level of privacy indicates that thevideo data is more sensitive, from an integrity point of view, thanvideo data having a lower level of privacy. In particular, differentportions of an object may be associated with different levels ofprivacy. For example, the plurality of objects 502-1, 502-2, 502-3 shownin FIG. 5A may have different portions being associated with differentprivacy levels. For example, an object may have a first portion 504-1,504-2, 504-3 corresponding to a first level of privacy, and a secondportion 506-1, 506-2, 506-3 corresponding to a second, higher level ofprivacy. In the illustrated example of FIG. 5A, the object 502-1, 502-2,502-3 is a person, the first portion of the objects 504-1, 504-2, 504-3corresponds to the body of the person, and the second portion 506-1,506-2, 506-3 of the objects corresponds to the head of the person. Thehead of a person are hence in this example associated with a higherprivacy level than the rest of the body of the person. It is understoodthat an object may have more than two portions having different privacylevels. By way of example, the hands of the person may be considered athird portion of the person being associated with a privacy level whichis different from that of the body and the head of the person. It isfurther understood that the objects are not limited to persons. Inanother example, an object could be a car, and the license plate of thecar may be given a higher privacy level than the rest of the car.Further, the windows of the cars may be considered a third portion ofthe car being associated with a privacy level which is different fromthat of the license plate and any other portion of the car.

The video streams 106, 108, 110 hence include video data depictingobject portions being associated with different levels of privacy.Particularly, the video stream 106 only includes video data depictingthe background and not depicting any portion of the one or more objects502-1, 502-2, 502-3. The privacy level of the video stream 106 thereforecorrespond to the privacy level of the background, i.e., a privacy levelwhich is lower than both the first and the second privacy level. Thevideo stream 108 includes video data depicting object portions 504-1,504-2, 504-3 being associated with the first privacy level. The privacylevel of the video stream 106 therefore correspond to the first privacylevel. The video stream 110 includes video data depicting objectportions 506-1, 506-2, 506-3 being associated with the second, higher,privacy level. The privacy level of the video stream 110 thereforecorrespond to the second privacy level. The privacy levels of thestreams 106, 108, 110 thus form a hierarchy of privacy levels, rangingfrom a lower to a highest privacy level.

The video streams 106, 108, 110 may be encrypted with differentencryption keys depending on the privacy level of the object portionsbeing depicted therein. For example, the video stream 108 may beencrypted using an encryption key associated with the first privacylevel, and the video stream 110 may be encrypted using an encryption keyassociated with the second, higher, privacy level. The video stream 106may be encrypted with an encryption key associated with a privacy levelof the background. Alternatively, the video stream 106 may not beencrypted at all.

The purpose of the encryption is to control access to the video data inthe video streams 106, 108, 110. In particular, in order to access videodata in a video stream 106, 108, 110, the video stream must first bedecrypted. Thus, only a user having access to the correct decryption keycan decrypt a video stream and access the video data therein. A user'saccess to decryption keys, and thereby the access to video data ofdifferent privacy levels, can be controlled via permissions. Forexample, a user may have a permission to access video data depictingobject portions of the first privacy level, but not the second, higher,privacy level. Such a permission may grant access to a decryption keythat can be used to decrypt the partially masked video stream 108, butnot to a decryption key that can be used to decrypt the non-masked videostream 110. Another user may have a permission to access video datadepicting object portions of the second, higher, privacy level. Such ahigher permission may grant access to a decryption key that can be usedto decrypt the non-masked video stream 110 associated with the secondprivacy level. Accordingly, a higher user permission may be required todecrypt the non-masked video stream 110 than the partially masked videostream 108. Further, a lowest user permission may be required to decryptthe fully masked video stream 106 in case it is encrypted. It is furtherunderstood that a permission that grants access to decryption keys fordecrypting a video stream associated with a certain privacy level alsogrants access to decryption keys for decryption video streams associatedwith privacy levels being lower than that certain level. By way ofexample, a user having a permission to access the video stream 110depicting the heads of the persons also has the permission to access thevideo stream 108 depicting the bodies of the persons and the videostream 106 depicting the background of the objects.

FIG. 2 illustrates the device 104 in more detail. The device 104includes a video data access component 202, a display component 204, apermission checking component 206, and a user input receiving component208. The video data access component may include a decryption component202 a and a video stream generating component 202 b.

The operation of the device 104 when executing a method for partiallyunmasking an object in a video stream will now be explained in moredetail with reference to FIGS. 1, 2, 5, 7 and the flowchart of FIG. 6.

In step S02, the display component 204 displays the video stream 106,referred to herein as the first video stream, in which one or moreobjects 502-1, 502-2, 502-3 are covered by privacy masks. FIG. 7Aillustrates a snapshot of the video stream 106 as displayed by thedisplay component 204. For example, the display component 204 may causethe first video stream 106 to be shown on a display to a user currentlybeing logged into the device 104. In case the first video stream 106 isencrypted, the decryption component 202 a may first decrypt the firstvideo stream.

In step S04 the user input component 208 receives a user input regardinga selection of an object being covered by a privacy mask in the firstvideo stream 106. The user input is received from a user currently beinglogged into the device 104. By way of example, object 502-1 shown in thesnapshot of FIG. 7A may be selected by the user input. The selectedobject has a first portion 504-1 being associated with a first privacylevel and a second portion 506-1 being associated with a second, higher,privacy level. The user input may be indicative of an image region whichincludes the selected object 502-1 in the first video stream 106. In oneexample, the user input may specify the outline of such an image region,e.g., by drawing a rectangle around the selected object 502-1 in thevideo stream shown on the display. Accordingly, the image region is notrestricted to only include the selected object 502-1, but portions ofthe background and of other objects may also be included within theimage region. In this example, objects that fall within the indicatedimage region will be partially unmasked as further described below. Itis noted that the selected object(s) may, over time, move out of theindicated image region. In that case, the user may via a further userinput move the image region to again include the selected object(s). Inthat way, it is possible to adjust the image region over time to keeppartially unmasking the selected object(s). Alternatively, the movementof the selected object(s) may be derived from object tracking metadatawhich assigns object identifiers to privacy masked image regions in thevideo and keeps them updated over time. Thus, if the image regionindicated by the user input includes one or more objects, their objectidentifiers may be derived from the object tracking metadata. Themovements of the objects with those identifiers may then be followed viathe object tracking metadata. In that way, image regions correspondingto the selected objects may be tracked over time. This enables keepingthe selected objects partially unmasked over time.

In another example, the user input may specify an image position withinan object being covered by a privacy mask in the first video stream 106.For example, the user may click on an object to select that object. Inthe latter example, metadata included in the first video stream 106 maybe used in order to derive what image region in the video stream 106corresponds to the selected object 502-1. Further, object trackingmetadata enables following the image regions corresponding to theselected object(s) over time. FIG. 7A shows an image region 702-1 whichincludes the selected object 502-1. It is further understood that theuser input may concern a selection of more than one object being coveredby a privacy mask. For example, the user may select two or all objectsdepicted in the first video stream 106. In that case, the image regionwill instead include all selected objects.

The received user input may further be logged by the device 104. Forexample, a received user input may be logged together with informationregarding the indicated image region or an indication regarding whichobjects were selected. Further, the number of selected objects may belogged. In that way, a user's action may be tracked to identify possibleabuse of the method. For example, it would be possible to identify if auser deviates from his instructions and indicates that he wants thewhole images of the video partially unmasked for long periods of time.

Responsive to the user input, the permission checking component 206checks in step S06 whether a permission associated with the user inputgrants access to video data depicting object portions associated withthe first privacy level. In more detail, the permission checkingcomponent 206 may check the permissions of the logged-in user. This mayinclude checking whether the logged-in user has access right to adecryption key that may be used to decrypt video data depicting objectportions of the first privacy level. Specifically, it may be checkedthat the logged-in user has access right to a decryption key fordecrypting the partially masked video stream 108. For example, thepermission checking component 206 may check whether the logged-in useris part of a user group, such as an active directory, that has right toaccess video data depicting object portions associated with the firstprivacy level. Members of such a user group may, in particular, haveaccess right to a decryption key that may be used decrypt the videostream 108.

In case the permission checking component 206 comes to the conclusionthat the permission associated with the user input does not grant accessto video data of the first privacy level, it will instruct the videodata access component 202 to not give access to any video data of thefirst privacy level. As a result, the display component 204 will keepdisplaying the first video stream 106 to the user.

In case the permission checking component 206 finds that the permissionassociated with the user input grants access to video data of the firstprivacy level, it instructs the video data access component 202 to giveaccess to video data depicting the first portion 504-1 of the selectedobject 502-1. Generally, the video data access component 202 controlswhich video data is accessed and displayed by the display component 204.The video data access component 402 controls access to video data by wayof decryption. In particular, if a permission associated with the userinput grants access to video data depicting certain object portions, thevideo data access component 202 may decrypt such video data.

Responsive to the instruction from the permission checking component206, the video data access component 202 proceeds in step S08 to giveaccess video data depicting the first portion 504-1 of the selectedobject 502-1. In more detail, the decryption component 202 a may proceedto decrypt the partially masked video stream 108 using the decryptionkey that the logged-in user has permission to access to produce adecrypted video stream 108′. A snap-shot of the decrypted video stream108′ is shown in FIG. 7B. As shown, object portions 504-1, 504-2, 504-3of the first privacy level (the bodies) of the objects are visible inthe decrypted video stream 108, while object portions 506-1, 506-2,506-3 of the second privacy level (the heads) are covered by privacymasks. The decrypted video stream 108′ may then be input to the videostream generating component 202 b which merges it with the first videostream 106 to generate a second video stream 112. A snapshot of thesecond video stream 112 is shown in FIG. 7C. The second video stream 112differs from the first video stream 106 in that it includes video datadepicting the first portion 504-1 of the selected object 502-1, whilethe second portion 506-1 of the selected object 502-1 is still coveredby a privacy mask. In other aspects the second video stream 112 isidentical to the first video stream 106. Specifically, the othernon-selected objects 502-2 and 502-3 are still covered by privacy masks.

In order to generate the second video stream 112, the video streamgenerating component 202 a may combine video data from the first videostream 106 with video data from the decrypted video stream 108′. Inparticular, in the image region 702-1 corresponding to the selectedobject 502-1 it may use video data from the decrypted video stream 108′,while video data from the first video stream 106 is used outside of theimage region 702-1. For example, the video stream generating component210 may extract video data from the image portion 702-1 corresponding tothe selected object 502-1 from the decrypted video stream 108′. The soextracted video data may then be used to replace the corresponding imageportion 702-1 in the first video stream 106. This process may berepeated for each video frame to produce the video stream 112.

In step S10, the display component 204 then displays the second videostream 112. Accordingly, responsive to a user input that selects anobject in the displayed first video stream 106, a permitted user ispresented with a second video stream 112 in which the selected object ispartially unmasked to reveal video data depicting object portions havingthe first level of privacy, but not object portions having the second,higher, level of privacy.

In step S06, the permission checking component 206 checked whether apermission associated with the user input granted access to video datadepicting object portions of the first privacy level. If that was thecase, the selected object 502-1 in the first video stream 106 waspartially unmasked to reveal video data depicting its object portion504-1 of the first privacy level. In some cases, the permission of theuser input may further grant access to video data depicting objectportions 506-1, 506-2, 506-3 of the second, higher, privacy level. Inthose cases, the user may, if permitted, further be presented with avideo stream where the selected object is further unmasked to alsoreveal video data depicting the object portion of the second privacylevel. For example, the user may provide a further user input toindicate that he desires to further unmask the selected object 502-1. Inresponse to such a further user input, the permission checking component206 may check whether the permission associated with either the first orthe further user-input grants access to video data depicting objectportions 506-1, 506-2, 506-3 of the second privacy level. The check maybe performed in the same way as described in connection to step S06above, e.g. by checking that the currently logged-in user has right toaccess a decryption key corresponding to the video stream 110 depictingobject portions of the second privacy level. If there is such apermission, the permission checking component 206 may instruct the videodata access component 202 to give access video data depicting the secondportion 506-1 of the selected object 502-1. The video access component206 may then proceed to give access to the video data. For example, thedecryption component 202 a may decrypt the video stream 110 using theappropriate decryption key that the user has the permission to access.The decrypted version 110′ of video stream 110 may then be input to thevideo stream generating component 202 a. By merging the decrypted videostream 110′ with the first video stream 106, a third video stream 114which includes video data depicting both the first portion 504-1 and thesecond portion 506-1 of the selected object 502-1 may be generated. Themerging can be performed according to the same principles as describedabove in connection to the generation of the second video stream 112. Asnapshot of the video stream 114 is shown in FIG. 7D. As can be seentherein the video stream 114 differs from the first video stream 106 inthat it includes video data depicting both the first portion of theobject and the video data depicting the second portion of the object. Inother aspects it is identical to the first video stream 106. In caseswhere the video stream 110′ only includes video data depicting objectportions of the first privacy level 506-1, 506-2, 506-3, the decryptedvideo stream 110′ may be merged with both the first video stream 106 andthe video stream 108′. For example, video data depicting the firstobject portion 504-1 of the selected object 502-1 may be collected fromthe video stream 108′, video data depicting the second object portion506-1 of the selected object 502-1 may be collected from the videostream 110′, and video data depicting the remainder of the scene may becollected from the first video stream 106. The so collected video datamay then be combined to produce the video stream 114. The video stream114 may then be displayed by the display component 204. Accordingly,responsive to a further user input, a permitted user is presented with avideo stream 114 in which the selected object is unmasked to revealvideo data depicting object portions of the first level of privacy aswell as video data depicting object portions of the second level ofprivacy.

FIG. 3 illustrate an alternative system 300 in which example embodimentsmay be implemented. The system 300 comprises a camera 302 and a device304 for partially unmasking an object in a video stream. The device 304may form part of a video management system via which a user may managevideo captured by the camera 302. The system 300 differs from the system100 in that the camera 302 does not perform any privacy masking. Thecamera 302 instead outputs a video stream 310 which does not include anyprivacy masks. A snapshot of the video stream 310 is shown in FIG. 8A.As will be described in more detail below, any privacy masking ofobjects depicted in the video stream 310 takes place in the device 304.

FIG. 4 illustrates the device 304 in more detail. The device 304includes a video data access component 402, a display component 404, apermission checking component 406, and a user input receiving component408. The display component 404, the permission checking component 406,and the user input receiving component 408 operate in the same way asthe corresponding components of the device 104. However, the video dataaccess component 402 differs from the corresponding component of thedevice 104. The video data access component 402 may include an objectdetection component 402 a, and a privacy masking component 402 b.

The operation of the device 304 when executing a method for partiallyunmasking an object in a video stream will now be explained in moredetail with reference to FIGS. 3, 4, 8 and the flowchart of FIG. 6.

The device 304 receives a video stream 310 as input. The video stream310 does not include any privacy masks. Thus, and as shown in FIG. 8A,video data depicting all objects 502-1, 502-2, 502-3 is included in theinput video stream 310.

The video stream 310 is received by the video data access component 402.The video data access component 402 controls access to the video dataincluded in the video stream 310. Particularly, it controls which videodata is accessed by the display component 404. As will be furtherexplained below, the access control is based on user permissions. Thevideo data access component 402 controls access to video data by usingprivacy masking. In particular, it may cover objects or object portionsdepicted in the video stream 310 by privacy masks in order to preventthe video data depicting those object or object portions from beingaccessed.

Initially, the video data access component 402 performs privacy maskingto cover the one or more objects 502-1, 502-2, 502-3 depicted in thevideo stream 310. Thus, at this stage, the video access component 402does not give access to video data depicting the one or more objects502-1, 502-2, 502-3. The resulting video stream is output from the videoaccess component 402 as a first video stream 306. A snapshot of thefirst video stream 306 is shown in FIG. 8B. In step S02, the first videostream 306 is displayed by the display component 404.

The privacy masking may take place in two steps; first the objectdetection component 402 a may perform object detection to detect theobjects 502-1, 502-2, 502-3 in the video stream 310, and then theprivacy masking component 402 b may cover the detected objects 502-1,502-2, 502-3 with privacy masks. In order to detect objects, the objectdetection component 402 a may use object detection algorithms that areknown in the art. This includes object detection algorithms that arebased on machine learning, such as deep learning-based algorithms. Inthe example of FIG. 8, it may use algorithms that detect persons. Theobject detection component 402 a may further detect object portions. Forinstance, it may detect the object portions 506-1, 506-2, 506-3 (heads)being associated with the second privacy level. For that purpose, it mayuse a detection algorithm adapted to detect such object portions. Forinstance, it may use a head detection algorithm to detect the heads ofthe persons. Generally, what types of objects and object portions todetect are typically predefined and known to the object detectioncomponent 402 a.

The object detection component 402 a may output information 403 beingindicative of the detected objects and object portions. The information403 may indicate image regions in the video stream 310 where objectsand/or object portions are detected. It may also indicate the type ofobject or object portion, such as whether it is a person, or a head.

The information 403 is then used by the privacy masking component 402 bwhen adding privacy masks to the input video stream 310. For example, itmay be used in order to cover image regions indicated in the information403 by privacy masks. Specifically, to generate the first video stream306, the privacy masking component 402 b may cover image regions ofobjects being of the type “persons”. The privacy masks may be of anytype that protect the integrity of the object or object portion. Forexample, it may be a contour mask, a rectangular mask, a pixelatedprivacy mask, a Sobel mask, a solid privacy mask, etc.

The privacy masking component 402 b may further keep a record of privacylevels associated with different types of objects or object portions. Inthe illustrated example, the record may specify that the object portiontype “body” has a first privacy level, while the object portion type“head” has a second, higher, privacy level. This enables the privacymasking component 402 b to selectively privacy mask object portionsbased on their associated privacy levels.

In step S04, a user input regarding a selection of one or more of theobjects 502-1, 502-2, 502-3 being covered by a privacy mask in the firstvideo stream 306 is received via user input receiving component 408. Forexample, the user input may select object 502-1. The device 304 maygenerally perform steps S04 and S06 in the same manner as the previouslydescribed device 104.

Responsive to that user input, the permission checking component 406proceeds in step S06 to check whether a permission of the user inputgrants access to video data depicting object portions of the firstprivacy level. For example, the permission checking component 206 maycheck whether the logged-in user is part of a user group, such as anactive directory, that has right to access video data depicting objectportions associated with the first privacy level.

In case the user input is associated with a user permission that grantsaccess to video data depicting object portions 504-1, 504- 2, 504-3 ofthe first privacy level, the permission checking component 406 mayinstruct the video data access component 402 to give access to suchvideo data for the selected object 502-1. Responsive to the instruction,in step S08, the video data access component 402 gives the displaycomponent 404 access to video data depicting the object portion 504-1 ofthe first privacy level for the selected object 502-1.

In more detail, the video data access component 402 may generate asecond video stream 312 which is forwarded to the display component 404.A snapshot of the second video stream 312 is shown in FIG. 8C. As isseen therein, the second video stream 312 differs from the first videostream 306 in that it includes the video data depicting the firstportion 504-1 of the selected object 502-1, while the second portion506-1 of the object is still covered by a privacy mask. The second videostream 312 may be generated by the privacy masking component 402 b fromthe input video stream 310 and with knowledge of the information 403from the object detecting component 402 a and information regarding theselected object 502-1 from the user input component 408. Specifically,the privacy masking component 402 b may for the selected object 502-1only cover object portions 506-1 of a type which is associated with ahigher privacy level than the first privacy level. For non-selectedobjects 502-2, 502-3, the privacy masking component 402 b keeps coveringthe whole objects with privacy masks. Thus, as shown in FIG. 8C, theselected object 502-1 is partially unmasked, while the other objectsremain fully masked.

As previously explained with respect to the device 104, the user inputmay further grant access to video data depicting object portions 506-1,506-2, 506-3 of the second, higher, privacy level. In those cases, theuser may, if permitted, further be presented with a video stream wherethe selected object is further unmasked to also reveal video datadepicting the object portion of the second privacy level. For example,the user may provide a further user input to indicate that he desires tofurther unmask the selected object 502-1. In response to such a furtheruser input, the permission checking component 406 may check whether thepermission associated with either the first or the further user-inputgrants access to video data depicting object portions 506-1, 506-2,506-3 of the second privacy level. The check may be performed in thesame way as described in connection to step S06 above, e.g. by checkingthat the currently logged-in is part of a user group having permissionto view video depicting object portion of the second privacy level. Ifthere is such a permission, the permission checking component 406 mayinstruct the video data access component 402 to access video datadepicting the second portion 506-1 of the selected object 502-1. Thevideo data access component 402 may then proceed to give the displaycomponent 404 access to the video data. In particular, the privacymasking component 402 b may generate a third video stream 314. The thirdvideo stream 314 differs from the first video stream 306 in that itincludes video data depicting both the first portion 504-1 of theselected object 502-1 and the video data depicting the second portion506-1 of the selected object 502-1. In other aspects it is identical tothe first video stream 306. A snapshot of the third video stream isshown in FIG. 8D. The privacy masking component 402 b may generate thethird video stream 314 from the input video stream 310 by covering anynon-selected object 502-2, 502-3 by a privacy mask, and not coveringobject portions of the first and second privacy level of the selectedobject 502-1. In the illustrated example, the selected object 502-1 willhence be fully uncovered. The video stream 314 may then be displayed bythe display component 204. Accordingly, responsive to a further userinput, a permitted user is presented with a video stream 314 in whichthe selected object further unmasked to reveal video data depictingobject portions of the first level of privacy as well as video datadepicting object portions of the second level of privacy.

The devices 104, 204 comprises various components 202, 202 a, 202 a,204, 206, 208, 402, 402 a, 402 b, 404, 406, 408 which are configured toimplement the functionality of the respective device 104, 204. Inparticular, each illustrated component corresponds to a functionality ofthe respective device 104, 204. Generally, the devices 104, 204 maycomprise circuitry which is configured to implement the components 202,202 a, 202 a, 204, 206, 208, 402, 402 a, 402 b, 404, 406, 408 and, morespecifically, their functionality. Thus, the circuitry may be said to beconfigured to execute the corresponding functionality.

In a hardware implementation, each of the components 202, 202 a, 202 a,204, 206, 208, 402, 402 a, 402 b, 404, 406, 408 may correspond tocircuitry which is dedicated and specifically designed to provide thefunctionality of the component. The circuitry may be in the form of oneor more integrated circuits, such as one or more application specificintegrated circuits or one or more field-programmable gate arrays. Byway of example, the display component 204 may thus comprise circuitrywhich, when in use, displays a video stream.

In a software implementation, the circuitry may instead be in the formof a processor, such as a microprocessor, which in association withcomputer code instructions stored on a (non-transitory)computer-readable medium, such as a non-volatile memory, causes thedevice 104, 204 to carry out any method disclosed herein. Examples ofnon-volatile memory include read-only memory, flash memory,ferroelectric RAM, magnetic computer storage devices, optical discs, andthe like. In a software case, the components 202, 202 a, 202 a, 204,206, 208, 402, 402 a, 402 b, 404, 406, 408 may thus each correspond to aportion of computer code instructions stored on the computer-readablemedium, that, when executed by the processor, causes the device 104, 204to carry out the functionality of the component.

It is to be understood that it is also possible to have a combination ofa hardware and a software implementation, meaning that the functionalityof some of the components 202, 202 a, 202 a, 204, 206, 208, 402, 402 a,402 b, 404, 406, 408 are implemented in hardware and others in software.

It will be appreciated that a person skilled in the art can modify theabove-described embodiments in many ways and still use the advantages asshown in the embodiments above. For example, in the above a firstexample system in which object detection, privacy masking and encryptiontakes place in the camera, and a second example system in objectdetection and privacy masking takes place in a video management systemwere described. However, it is understood that example systems where oneor more of these tasks are distributed over one or more devices whichare separate from the camera and the video managements system areequally possible. In a first group of examples, the camera may capturevideo which is sent to such a separate device for object detection,privacy masking and encryption, before the result is forwarded to thevideo management system. The separate device may be a cloud-baseddevice. If the camera is an analog camera, the separate device may be anencoder used to encode the video. The object detection may be performedby a first separate device, while the privacy masking and encryption maybe performed by a second separate device. In a second group of examples,the camera captures video which is sent to a separate device for objectdetection, while privacy masking based on the object detection is stillcarried out in the video management system. The separate deviceperforming the object detection may be a cloud-based device. In a thirdgroup of examples, the camera captures video which is forwarded to afirst separate device for object detection, and then to a secondseparate device which performs privacy masking and access control bychecking user permissions. In that case, the video management system ismerely used as a user interface for displaying video and receiving userinput. Thus, the teachings should not be limited to the shownembodiments but should only be defined by the appended claims.Additionally, as the skilled person understands, the shown embodimentsmay be combined.

1. A method for partially unmasking a selected object in a video stream,comprising: displaying a first video stream in which a plurality ofobjects are covered by privacy masks, wherein each of the plurality ofobjects has a first portion being associated with a first privacy level,and a second portion being associated with a second, higher, privacylevel, receiving user input regarding a selection of one object amongthe plurality of objects being covered by privacy masks in the firstvideo stream, checking that a permission associated with the user inputgrants access to video data depicting object portions being associatedwith the first privacy level, giving access to video data depicting thefirst portion of the selected object, and displaying a second videostream which differs from the first video stream in that it includes thevideo data depicting the first portion of the selected object, while thesecond portion of the selected object is still covered by a privacymask.
 2. The method of claim 1, wherein the video data depicting thefirst portion of the selected object is encrypted, and wherein access tothe video data depicting the first portion of the selected object isgiven by decrypting the video data depicting the first portion of theselected object using a first decryption key associated with thepermission that grants access to video data depicting object portionsbeing associated with the first privacy level.
 3. The method of claim 1,further comprising: generating the second video stream by merging thefirst video stream with the video data depicting the first portion ofthe selected object.
 4. The method of claim 1, wherein the video datadepicting the first portion of the selected object is accessed from avideo stream in which object portions associated with a first privacylevel are visible and object portions associated with the second privacylevel are covered by privacy masks.
 5. The method of claim 4, whereinthe second video stream is generated by replacing an image region whichincludes the selected object being covered by a privacy mask in thefirst video stream by a corresponding image region in the video streamin which object portions associated with a first privacy level arevisible and object portions associated with the second privacy level arecovered by privacy masks.
 6. The method of claim 1, wherein the userinput regarding a selection of one object among the plurality of objectsis indicative of an image region in the first video stream, the imageregion including the selected object being covered by a privacy mask inthe first video stream.
 7. The method of claim 1, wherein the firstvideo stream includes metadata being indicative of image regions in thefirst video stream being covered by privacy masks, wherein the userinput regarding a selection of one object among the plurality of objectsis indicative of an image position within one of the image regions. 8.The method of claim 1, further comprising: checking that the permissionassociated with the user input further grants access to video datadepicting object portions being associated with the second privacylevel, giving access to video data depicting the second portion of theobject, and displaying a third video stream which differs from the firstvideo stream in that it includes both the video data depicting the firstportion of the selected object and the video data depicting the secondportion of the selected object.
 9. The method of claim 8, wherein thestep of displaying the third video stream is made in response to afurther user input.
 10. The method of claim 8, wherein the video datadepicting the second portion of the selected object is encrypted, andwherein access to the video data depicting the second portion of theselected object is given by decrypting the video data depicting thesecond portion of the selected object using a second decryption keyassociated with the permission that further grants access to video datadepicting object portions being associated with the second privacylevel.
 11. The method of claim 6, further comprising: generating thethird video stream by merging the first video stream with the video datadepicting the first portion of the selected object and the video datadepicting the second portion of the selected object.
 12. The method ofclaim 8, wherein the video data depicting the second portion of theselected object is accessed from a video stream in which object portionsassociated with at least the second privacy level are visible.
 13. Themethod of claim 12, wherein the video data depicting the second portionof the selected object is accessed from a video stream in which onlyobject portions associated with the second privacy level are visible.14. A device for partially unmasking a selected object in a videostream, comprising circuitry configured to execute: a display functionconfigured to display a first video stream in which a plurality ofobjects are covered by privacy masks, wherein each of the plurality ofobjects has a first portion being associated with a first privacy level,and a second portion being associated with a second, higher, privacylevel, a user input receiving function configured to receive a userinput regarding a selection of one object among the plurality of objectsbeing covered by a privacy mask in the first video stream, a permissionchecking function configured to check that a permission associated withthe user input grants access to video data depicting object portionsbeing associated with the first privacy level, a video data accessingfunction configured to give access to video data depicting the firstportion of the selected object, and a display function configured todisplay a second video stream which differs from the first video streamin that it includes the video data depicting the first portion of theselected object, while the second portion of the selected object isstill covered by a privacy mask.
 15. A non-transitory computer-readablestorage medium comprising computer-code instructions adapted to carryout a method, when executed by a device having processing capability,comprising: displaying a first video stream in which a plurality ofobjects are covered by privacy masks, wherein each of the plurality ofobjects has a first portion being associated with a first privacy level,and a second portion being associated with a second, higher, privacylevel, receiving user input regarding a selection of one object amongthe plurality of objects being covered by privacy masks in the firstvideo stream, checking that a permission associated with the user inputgrants access to video data depicting object portions being associatedwith the first privacy level, giving access to video data depicting thefirst portion of the selected object, and displaying a second videostream which differs from the first video stream in that it includes thevideo data depicting the first portion of the selected object, while thesecond portion of the selected object is still covered by a privacymask.